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Program description 


Talent Development High Schools is a school reform model for communities”— including ninth-grade academies for first-year 

restructuring large high schools with persistent attendance and students and career academies for students in upper grades— to 

discipline problems, poor student achievement, and high drop- reduce student isolation and anonymity. It also emphasizes high 

out rates. The model includes both structural and curriculum academic standards and provides all students with a college- 

reforms. It calls for schools to reorganize into small “learning preparatory academic sequence. 



Research 



One study of Talent Development High Schools met the What 
Works Clearinghouse (WWC) evidence standards with reserva- 
tions. The quasi-experimental research design included multiple 
cohorts of entering ninth-grade students from 11 Philadelphia 
high schools— five Talent Development High Schools and six 



matched comparison schools.'' The WWC considers the extent 
of evidence for Talent Development High Schools to be small for 
progressing in school. No studies that met the WWC evidence 
standards with or without reservations addressed staying in 
school or completing school. 



Effectiveness Talent Development High Schools was found to have potentially positive effects on progressing in school. 





Staying in schooi 


Progressing in schooi 


Compieting schooi 


Rating of effectiveness 


na 


Potentially positive effects 


na 


Improvement index^ 


na 


Average: +7 percentile points 


na 






Range: +6 to +8 percentile points 





na = not applicable 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average improvement index for all findings across the study. 
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Additional program 
information 
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Developer and contact 

Talent Development High Schools was developed by The Center 
for Research on the Education of Students Placed At Risk 
(CRESPAR), housed at Johns Hopkins University’s Center for 
the Social Organization of Schools (CSOS). Information on the 
model’s history and current resources for program implementa- 
tion are available from CSOS at www.csos.ihu.edu/tdhs . 

Scope of use 

Talent Development High Schools was initiated in 1994 through 
a partnership between CRESPAR and Patterson High School 
in Baltimore, Maryland. CSOS reports that, as of March 2007, 

43 districts in 15 states were operating schools using the Talent 
Development High Schools model in full or in part. 

Description of intervention 

Talent Development High Schools is a school reform model for 
restructuring large high schools facing serious problems with 
attendance, discipline, student achievement, and dropping 
out. To address these problems and to prepare all students 
for post-secondary education and employment, the model 
introduces both structural and curriculum reforms. To reduce 
student anonymity and isolation. Talent Development High 
Schools reorganizes high schools into smaller learning com- 
munities, including a ninth-grade academy, career academies 
for the upper grades, and an after-hours school for students 
with serious discipline problems. The ninth-grade academy is 
a self-contained school-within-a-school for first-year students, 
taught by a team of four to five teachers. Career academies for 
the upper grades, self-contained groups of about 300 students 
organized around career themes, have their own teaching staff 
and management. The “Twilight School,” an after-hours program 
for students with serious attendance or discipline problems, 
provides small classes and extensive support services. 



Curriculum reforms, complementing the structural changes, 
address low student expectations and poor academic prepara- 
tion, which the model views as root causes of dropping out. To 
increase expectations for student achievement. Talent Develop- 
ment High Schools provides a college-preparatory academic 
sequence for all students. The program provides “double dose” 
mathematics and English courses for ninth and tenth graders. 

The first semester of “double dose” courses is remedial English 
or math; the second semester is the district-mandated course, a 
full-credit (and typically year-long) course covered in one semester 
of daily 90-minute sessions. In addition, as part of the ninth-grade 
academy, all first-year students complete a one-semester seminar 
that teaches strategies for meeting the increased academic 
demands of high school. 

To address the challenges of implementing large-scale school 
reform. Talent Development High Schools emphasizes ongoing 
technical assistance and professional development for staff. 

Each school is assigned a team of curriculum coaches trained by 
CSOS to work with school staff to implement the model. In addi- 
tion, CSOS sponsors annual conferences for Talent Development 
High Schools staff. 

Cost 

According to the CSOS, the additional cost of operating Talent 
Development High Schools (above and beyond the cost of 
continuing to operate their traditional high school model) is about 
$350 per student per year. This estimate includes the costs of 
curriculum materials and ongoing technical assistance. CSOS 
indicates that school districts may have additional expenses if the 
shift to block scheduling and the implementation of the academy 
model requires them to hire additional staff. In some cases, 
school districts may also incur additional costs if they need to 
renovate their facilities so that the ninth-grade academy and the 
career academies can be housed in distinct parts of the building. 
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Research 



Effectiveness 
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The WWC reviewed four studies of the effectiveness of Talent 
Development High Schools. One study (Kemple, Heriihy, & 

Smith, 2005) used a quasi-experimental research design and 
met WWC evidence standards with reservations. The other three 
studies did not meet WWC evidence screens. 

Met evidence standards with reservations 

The Kempie, Heriihy, & Smith (2005) study of Talent Development 
High Schools used a quasi-experimentai research design known 
as comparative interrupted time series analysis. The study 
focused on five Phiiadelphia high schoois that began imple- 
menting Talent Development High Schools between 1999 and 
2001. These schoois were matched to six simiiar Philadelphia 
high schoois that did not implement the program. The study 
compared the outcomes of ninth graders who entered Talent 
Development High Schools in the years immediateiy after the 
program was impiemented with the outcomes of ninth graders 



from these schools in the years just before program implementa- 
tion and the outcomes in the comparison schoois. The difference 
between outcomes before and after implementation in Talent 
Development High Schools and the comparison schools is the 
estimate of the program’s effects. 

Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
smaii or moderate to iarge (see the What Works Ciearinahouse 
Extent of Evidence Categorization Schemed The extent of evi- 
dence takes into account the number of studies and totai sample 
size across the studies that met WWC evidence standards with 
or without reservations.® 

The WWC considers the extent of evidence for Talent 
Development High Schools to be smaii for progressing in school. 
No studies that met WWC evidence standards with or without 
reservations addressed staying in school or completing school. 



Findings 

The WWC review of interventions for dropout prevention 
addresses student outcomes in three domains: staying in school, 
progressing in school, and completing school. For Kemple, 
Heriihy, & Smith (2005), WWC assessed outcomes only in the 
progressing in school domain."* 

Progressing In school. Kemple, Heriihy, & Smith (2005) found 
that students using Talent Development High Schools earned 
an average of 9.5 course credits over the first two years of high 
school, while comparison group students earned 8.6 course 



credits. In addition. Talent Development High Schools students 
were more likely to be promoted to tenth grade than comparison 
students (68% vs. 60%).® Both differences were statistically 
significant. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 



3. The Extent of Evidence Categorization was developed to teil readers how much evidence was used to determine the intervention rating, focusing on 
the number and size of studies. Additional factors associated with a related concept, external validity, such as students’ demographics and types of 
settings in which studies took place, are not taken into account for the categorization. 

4. The study also examined outcomes in the staying in school and completing school domains. However, these analyses did not meet WWC standards. 
Please see Appendix A1 for details. 

5. These comparison group means were not directly reported by Kemple, Heriihy, & Smith (2005) and were obtained by a simple transformation of the 
results provided in the report. See the WWC Talent Development High Schools Technical Appendices for more details. 
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Effectiveness (continued) 



The WWC found Talent 
Development High Schools 
to have potentially 
positive effects on 
progressing in school 



References 



design, the statistical significance of the findings,® the size of the parison conditions, and the consistency in findings across studies 

difference between participants in the intervention and the com- (see the WWC Intervention Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individuai 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see Technical Details 
of WWC-Conducted Computations) . The improvement index rep- 
resents the difference between the percentiie rank of the average 
student in the intervention condition versus the percentiie rank of 
the average student in the comparison condition. Uniike the rating 
of effectiveness, the improvement index is based entireiy on the 
size of the effect, regardiess of the statisticai significance of the 
effect, the study design, or the anaiyses. The improvement index 
can take on vaiues between -50 and +50, with positive numbers 
denoting resuits favorabie to the intervention group. 



The average improvement index for progressing in school is 
+7 percentile points based on the one study that passed WWC 
evidence screens, with a range of +6 to +8 percentile points 
across the findings. 

Summary 

The WWC reviewed four studies on Talent Development High 
Schools. One study met WWC standards with reservations; the 
remaining studies did not meet WWC evidence screens. Based 
on this one study, the WWC found potentially positive effects on 
progressing in schooi. The conciusions presented in this report 
may change as new research emerges. 



Met WWC evidence standards with reservations 

Kempie, J., Heriihy, C., & Smith, T. (2005). Making progress 
toward graduation: Evidence from the Talent Development 
High School model. New York: MDRC. 

Additional source: 

Kempie, J., & Heriihy, C. (2004). The Talent Development High 
School model: Context, components, and Initial impacts on 
ninth-grade students’ engagement and performance. New 
York: MDRC. 



Did not meet WWC evidence screens 

Baifanz, R., Legters, N., & Jordan, W. (2004). Catching up: 

Impact of the Talent Development ninth grade instructional 
interventions in reading and mathematics in high-poverty high 
schools. Baltimore: Johns Hopkins University, CRESPAR.^ 
McPartiand, J., Baifanz, R., Jordan, W., & Legters, N. (1998). 
Improving climate and achievement in a troubled urban high 
school through the Talent Development model. Journal of 
Education for Students Placed at Risk, 3(4), 337-361 .® 



6. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Kempie, Heriihy, & Smith (2005), the study authors 
had corrected for clustering, so no additional corrections were required. The WWC did, however, correct the statistical significance levels for multiple 
comparisons. 

7. The outcome measures are not relevant to this review. 

8. Lack of evidence of baseline equivalence: the study, which used a quasi-experimental design, did not establish that the comparison group was equiva- 
lent to the intervention group at baseline. 
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RefersncSS (continued) McPartland, J., Legters, N., Jordan, W., & McDill, E. L. (1996). The 

Talent Development High School: Early evidence of impact on 



school climate, attendance, and student development (Report 
No. 2). Baltimore: Johns Hopkins University, CRESPAR.® 
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For more information about specific studies and WWC caicuiations, piease see the WWC Talent Development 
High Schools Technicai Appendices . 



9. The study did not use a comparison group to assess relevant WWC outcomes. 
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Appendix 



Appendix A1 


Study characteristics: Kempie, Heriihy, & Smith, 2005 (quasi-experimentai design) 


Characteristic 


Description 


Study citation 


Kempie, J. J., Heriihy, C. M., & Smith, T.J. (2005). Making progress toward graduation: Evidence from the latent Deveiopment High Schooi modei. New York: MDRC. 


Participants 


The main analysis sample included first-time ninth-grade students^ from five high schools that began implementing Talent Deveiopment High Schools between 1999 and 2001 and 
six matched comparison high schools.^ Between two and four comparison schools were matched to each of the five intervention schools based on the racial/ethnic composition 
and promotion rates of the schools’ ninth-grade students (Kempie & Heriihy, 2004). A comparison school could be matched to multiple Talent Development High Schools. The study 
compared the outcomes of ninth graders who entered Talent Development High Schools in the three years immediately after the program was implemented with those of ninth graders 
from these schools in the three years just before program implementation and with the outcome differences over the same time period for the matched comparison schools.^ 

Many students selected for Talent Development High Schools had low test scores and were overage for their grade. More than three-quarters were African-American and 
about one in six were Hispanic. Poor attendance was common, with two-thirds missing at least 20% of scheduled school days during their ninth-grade year. In addition, many 
did not make regular progress toward graduation, with just half promoted to tenth grade at the end of their ninth-grade year. Students in the matched comparison schools 
were generally similar to Talent Development High Schools students on these characteristics (Kempie & Heriihy, 2004). 

The study examined three cohorts of students. Cohort 1 included students in the intervention and matched comparison schools who enrolled in the ninth grade during the first 
year of Talent Development High Schools implementation at the intervention schools. Similarly, Cohort 2 and Cohort 3 included students who were enrolled in the ninth grade 
during the second and third years of implementation, respectively. Given the fixed period for data collection, later cohorts had shorter follow-up periods. To ensure both an 
adequate follow-up and an adequate sample size for assessing program effectiveness, the WWC used second-year results based on Cohorts 1 and 2 to rate the effectiveness 
of Talent Development High Schools. Longer-term results based only on Cohort 1 and shorter-term results based on all three cohorts are reported in Appendix A4. 


Setting 


The impact study was conducted in 11 nonselective public high schools in Philadelphia. 


Intervention 


The Philadelphia public school district implemented the Talent Development High Schools model in seven high schools. The district began to roll out the program in 1998, with 
one or two high schools launching Talent Development High Schools eacU year over a five-year period. School administrators volunteered their schools as candidates for imple- 
menting the new program. To allow for adequate follow-up, the impact study excluded the two Philadelphia high schools that implemented Talent Development High Schools last. 

All the Philadelphia Talent Development High Schools created ninth-grade academies on a separate floor or wing of the building, which were taught by teams of four to five 
teachers. Each school introduced block scheduling with 80- to 90-minute class sessions, introducing "double dose” math and English courses for ninth and tenth graders. 
These double sections of English and math allowed students to both prepare for and take college preparatory classes over the course of one academic year. Six of the seven 
schools offered ‘Twilight School” for new or repeating ninth graders with serious attendance or discipline problems.'* 

The model for students in grades 10 through 12 centered around career academies, in which students were divided into smaller “learning communities” around a broad career 
interest and the curriculum was organized around a career theme. Many Philadelphia high schools already had career academies before Talent Development High Schools ms 
implemented, including many mn-Talent Development schools. The study authors concluded that “(i)t is likely, therefore, that the upper-grade experience of students in Talent 
Development schools did not greatly differ from that of students in mn-Talent Development schools" (Kempie, Heriihy, & Smith, 2005, p. 27). 

The study authors reported some variation in how the program was implemented across schools (Kempie, Heriihy, & Smith, 2005). In particular, they noted considerable varia- 
tion across the intervention schools in the amount of technical assistance and support schools received from the intervention developer, as well as the amount of intervention- 
specific training school staff received. 

(continued) 
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Appendix A1 Study characteristics: Kempie, Heriihy, & Smith, 2005 (quasi-experimentai design) (continued) 



Characteristic 


Description 


Comparison 


Matched comparison schools were nonselective Philadelphia high schools that did not Implement Talent Development High Schools. The authors compared the intervention 
group both with students in the comparison schools and with students who attended the intervention schools prior to the implementation of Talent Development High Schools. 


Primary outcomes 
and measurement 


Two relevant outcomes are included in this review: total credits earned and enrollment in the tenth grade by the end of the second year of high school (see Appendix A2 for 
more detailed descriptions of these outcome measures). 




The study also examined Talent Development High Schools’ ettects on attendance and student achievement. These outcomes are not included in this report because they do 
not fall within the three domains (staying in school, progressing in school, and completing school) examined by the WWC's review of dropout prevention interventions. Effects 
on the percentage of students who exited the school district were also estimated. However, the WWC had concerns about the validity of this measure and did not include it 
in the review.^ The study also examined the effects of Talent Development High Schools on graduation on the two earliest implementing schools. Since these results are only 
available for a small subset of the full research sample, they are not considered for the effectiveness rating and improvement index. 


Teacher training 


Teachers at Talent Development High Schools were regular teachers employed by the Philadelphia Public Schools. “Curriculum coaches” who had been trained by the intervention 
developer provided on-site technical assistance with implementing the Talent Development High Schools model. The developer also provided summer training institutes for staff. 



1 . The authors also examined the effects of Talent Development High Schools on students repeating the ninth grade. However, this review focuses only on the effects of the intervention on first- 
time ninth graders. 

2. The sample was restricted to students who attempted at least one course credit during the ninth grade and excluded special education students and English language learners. The authors did 
not report the number of students in the sample. 

3. To take into account the potential influence of changes in the composition of students attending the study schools, the authors used ordinary least squares regression to control for a set of 
student background characteristics (race, seventh-grade reading and math test scores, and whether the student had repeated a grade) when estimating impacts. 

4. Reports from the study do not indicate whether the one school that did not implement the “Twilight School” was among the five schools included in the impact analysis. 

5. In particular, the measure did not appear to adequately capture the full extent of dropping out because only about 2% of students were categorized as having dropped out during the first two 
years of high school. 
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Appendix A2 Outcome measures in the progressing in schooi domain 



Outcome measure 


Description 


Total credits earned by end 
of second year of high school 


This measure represents the cumulative number of course credits earned over the first two years of high school. These data were collected from individual studenfs’ school 
records obtained from fhe district. 


Enrolled in tenth grade 
by end of second year 
of high school 


This measure represents the percentage of students who were enrolled in the tenth grade by the end of the second year of high school. These data were collected from 
individual students’ school records obtained from the district. 
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Appendix A3 Summary of study findings inciuded in the rating for the progressing in schooi domain^ 









Authors’ findings from the study 














Mean outcome 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools)^ 


Talent 

Development 

High Schools Comparison 

groups group’’ 


Mean difference^ 
(Talent 
Development 
High Schools - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index® 



Kemple, Herlihy, & Smith, 2005 (quasi-experimental design)^ 



Total credits earned by 
end ot second year 


Cohorts 1 and 2 


11 


9.5 


8.6 


0.9 


0.16 


Statistically 

significant 


-f6 


Enrolled in tenth grade by 
end ot second year (%) 


Cohorts 1 and 2 


11 


68 


60 


8 


0.21 


Statistically 

significant 


-f8 



Domain average^" for progressing in school across all studies 0.18 Statistically +7 

significant 



1 . This appendix reports follow-up findings considered for the effectiveness rating and the improvement index. Kemple, Herlihy, & Smith (2005) examined Talent Development High Scftoo/s’ effects on three cohorts of entering ninth-grade 
students. Given the fixed period for data collection, the follow-up period varied by cohort. Cohort 1 had three years of follow-up, while Cohort 2 had two years, and Cohort 3 had only one year. To balance the benefits of assessing pro- 
gram effectiveness using a longer follow-up period with the benefits of assessing effectiveness using results from multiple cohorts of students, the WWC used results for Cohorts 1 and 2 measured at the end of the second year of high 
school. Follow-up findings from the end of the first year of high school (available for all three cohorts) and the third year of high school (available for Cohort 1 only) are not included in these ratings, but are reported in Appendix A4. 

2. Kemple, Herlihy, & Smith (2005) used individual student data, but did not report the number of students in the sample. 

3. The WWC calculated the cross-cohort average for each measure as the simple average of the mean outcomes for each cohort included in the measure. 

4 . Kemple, Herlihy, & Smith (2005) did not report these adjusted comparison group means and instead reported baseline to follow-up changes for both the intervention and comparison groups. The WWC generated the adjusted com- 
parison group means reported here using the following transformation: adjusted comparison group mean = follow-up comparison group mean + (baseline intervention group mean - baseline comparison group). Stated differently, 
the adjusted comparison group mean equals the follow-up intervention group mean minus the estimated impact because, under the comparative interrupted time-series technique used in Kemple, Herlihy, & Smith (2005), impacts are 
calculated as follows: impact = (follow-up intervention group mean - baseline intervention group mean) - (follow-up comparison group mean - baseline comparison group mean). 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. The effect size for the “total credits earned” outcome was reported by the study authors. The study authors calculated the effect size by dividing the impact at follow-up by the standard deviation of the outcome for all ninth-grade stu- 
dents in the Philadelphia school district's nonselective, comprehensive high schools from school years 1996/97 through 1998/99. The effect size for the dichotomous variable “enrolled in the tenth grade by the end of the second year” 
was computed using the Cox Index. For an explanation of the effect size calculations, see Technical Details of WWC-Conducted Computations . 

7. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

8. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and -i-50, with positive numbers denoting results favorable to the intervention group. 

9. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Con ducted C omputations for the formulas the WWC used to calculate statistical significance. In the case of Kemple, Herlihy, & Smith (2005), the study 
authors made appropriate corrections for clustering, so no additional corrections for clustering were necessary. The WWC corrected statistical significance levels for multiple comparisons. 

10. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size. 
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Appendix A4 Summary of shorter-term and longer-term findings for the progressing in school domain^ 









Authors’ findings from the study 














Mean outcome 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools)^ 


Talent 

Development 

High Schools Comparison 

groups group’’ 


Mean difference^ 
(Talent 
Development 
High Schools - 
comparison) 


Statistical 
significance^ 
Effect size® (at a = 0.05) 


Improvement 

index® 



Kemple, Herlihy, & Smith, 2005 (quasi-experimental design)^ 



Total credits earned by 
end ot tirst year 


Cohorts 1, 2, and 3 


11 


5.2 


4.5 


0.7 


0.25 


Statistically 

signiticant 


-FlO 


Total credits earned by 
end ot third year 


Cohort 1 


11 


13.2 


12.3 


0.9 


0.12 


ns 


-f5 


Enrolled in eieventh grade 
by end of third year 


Cohort 1 


11 


53 


47 


6 


0.16 


ns 


-f6 



1 . This appendix presents foiiow-up findings at the end of the first and third years of high schooi for the measures that faii in the progressing in schooi domain. The third-year findings were based oniy on students who began the ninth- 
grade during the first year of Ta/enf Dei/etopmenf impiementation (cohort 1). The first-year findings were based on aii three cohorts. The second-year findings used for effectiveness rating purposes are presented in Appendix A3. 

2. Kempie, Heriihy, & Smith (2005) used individuai student data, but did not report the number of students in the sampie. 

3. The WWC caicuiated the cross-cohort average for the “totai credits earned by end of first year” outcome as the simpie average of the mean outcomes for each cohort inciuded in the measure. 

4. Kempie, Heriihy, & Smith (2005) did not report these adjusted comparison group means and instead reported baseiine to foiiow-up changes for both the intervention and comparison groups. The WWC generated the adjusted compari- 
son group means reported here using the foiiowing transformation: adjusted comparison group mean = foiiow-up comparison group mean -i- (baseiine intervention group mean - baseiine comparison group). Stated differentiy, the ad- 
justed comparison group mean equais the foiiow-up intervention group mean minus the estimated impact, since, under the comparative interrupted time-series technique used in Kempie, Heriihy, & Smith (2005), impacts are caicuiated 
as foiiows: impact = (foiiow-up intervention group mean - baseiine intervention group mean) - (foiiow-up comparison group mean - baseiine comparison group mean). 

5. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

6. The effect size for the “totai credits earned” outcome was reported by the study authors. The study authors caicuiated the effect size by dividing the impact at foiiow-up by the standard deviation of the outcome for aii ninth-grade stu- 
dents in the Phiiadeiphia schooi district's nonseiective, comprehensive high schoois from schooi years 1996/97 through 1998/99. The effect size for the dichotomous variabie “enroiied in the tenth grade by the end of the second year” 
was computed using the Cox index. For an expianation of the effect size caicuiations, see Technicai Detaiis of WWC-Conducted Computations . 

7. Statisticai significance is the probabiiity that the difference between groups is a resuit of chance rather than a reai difference between the groups. 

8. The improvement index represents the difference between the percentiie rank of the average student in the intervention condition versus the percentiie rank of the average student in the comparison condition. The improvement index 
can take on vaiues between -50 and -t-50, with positive numbers denoting resuits favorabie to the intervention group. 

9. The ievei of statisticai significance was reported by the study authors or, where necessary, caicuiated by the WWC to correct for ciustering within ciassrooms or schoois. For an expianation, see the WWC Tutorial on Mismatch . See 
Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate the statistical significance. In the case of Kempie, Heriihy, & Smith (2005), the study authors made corrections for clustering and no ad- 
ditional corrections for clustering was necessary. 
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Appendix A5 Talent Development High Schools rating for the progressing in schooi domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of progressing in school, the WWC rated Talent Development High Schools as having potentiaiiy positive effects. It did not meet the criteria 
for positive effects because there was oniy one study and that study did not meet WWC evidence standards for a strong design. The remaining ratings (mixed effects, 
no discernible effects, potentially negative effects, and negative effects) were not considered because Talent Development High Schools was assigned the highest 
applicable rating. 



Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study of Talent Development High Schools demonstrated a statistically significant positive effect. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. No studies found statistically significant or substantively important negative effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Talent Development High Schools had only one study meeting WWC evidence standards. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies found statistically significant or substantively important negative effects in this domain. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A6 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schoois 


Sampie size 

Students 


Extent of evidence^ 


Staying in school 


0 


0 


0 


na 


Progressing in school 


1 


11 


nr 


Small 


Completing school 


0 


0 


0 


na 



na = not applicable/not studied 
nr = not reported 

1 . A rating of “moderate to large” requires at least two studies and two schools across studies in one domain, and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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